/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
I0528 11:54:45.992293 44434 gen_comm_id_helper.cc:181] Server listening on: 127.0.0.1:36563 successful.
I0528 11:54:46.086966 44434 nccl_context.cc:74] init nccl context nranks: 6 local rank: 5 gpu id: 9 ring id: 0
W0528 11:54:49.639700 44434 device_context.cc:404] Please NOTE: device: 9, GPU Compute Capability: 6.1, Driver API Version: 11.2, Runtime API Version: 10.2
W0528 11:54:49.646068 44434 device_context.cc:422] device: 9, cuDNN Version: 7.6.
/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
loading annotations into memory...
Done (t=0.42s)
creating index...
index created!
Illegal image file: datasets/camera_0525/mytrain/09269_2.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/10029_4.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/09171_6.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/18941_6.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/09171_8.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/09171_9.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/10029_10.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/09217_10.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/18941_10.jpg, and it will be ignored
Found an invalid bbox in annotations: im_id: 7609, area: 0.0 x1: 387.0, y1: 113.0, x2: 391.0, y2: 113.0.
Found an invalid bbox in annotations: im_id: 8206, area: 0.0 x1: 552.0, y1: 54.0, x2: 553.0, y2: 54.0.
Found an invalid bbox in annotations: im_id: 9019, area: 0.0 x1: 301.0, y1: 398.0, x2: 303.0, y2: 398.0.
Found an invalid bbox in annotations: im_id: 9019, area: 0.0 x1: 298.0, y1: 398.0, x2: 305.0, y2: 398.0.
Found an invalid bbox in annotations: im_id: 12643, area: 0.0 x1: 1091.0, y1: 26.0, x2: 1092.0, y2: 26.0.
Found an invalid bbox in annotations: im_id: 13527, area: 0.0 x1: 615.0, y1: 135.0, x2: 616.0, y2: 135.0.
ERROR:root:DataLoader reader thread raised an exception!
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 482, in _get_data
    data = self._data_queue.get(timeout=self._timeout)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/multiprocessing/queues.py", line 105, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 411, in _thread_loop
    batch = self._get_data()
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 498, in _get_data
    "pids: {}".format(len(failed_workers), pids))
RuntimeError: DataLoader 2 workers exit unexpectedly, pids: 20637, 20647

/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
I0528 13:03:07.364596 37741 gen_comm_id_helper.cc:181] Server listening on: 127.0.0.1:41399 successful.
I0528 13:03:07.525240 37741 nccl_context.cc:74] init nccl context nranks: 6 local rank: 5 gpu id: 9 ring id: 0


--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0   paddle::imperative::NCCLParallelContext::Init()
1   paddle::platform::NCCLCommContext::CreateNCCLComm(ncclUniqueId*, int, int, int, int)
2   ncclCommInitRank
3   paddle::framework::SignalHandle(char const*, int)
4   paddle::platform::GetCurrentTraceBackString[abi:cxx11]()

----------------------
Error Message Summary:
----------------------
FatalError: `Termination signal` is detected by the operating system.
  [TimeInfo: *** Aborted at 1622178191 (unix time) try "date -d @1622178191" if you are using GNU date ***]
  [SignalInfo: *** SIGTERM (@0x3e8000092e4) received by PID 37741 (TID 0x7faf66049740) from PID 37604 ***]

/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
I0528 13:05:18.093328 40718 gen_comm_id_helper.cc:181] Server listening on: 127.0.0.1:43131 successful.
I0528 13:05:18.565457 40718 nccl_context.cc:74] init nccl context nranks: 6 local rank: 5 gpu id: 9 ring id: 0
W0528 13:05:21.849432 40718 device_context.cc:404] Please NOTE: device: 9, GPU Compute Capability: 6.1, Driver API Version: 11.2, Runtime API Version: 10.2
W0528 13:05:21.856730 40718 device_context.cc:422] device: 9, cuDNN Version: 7.6.
/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
loading annotations into memory...
Done (t=0.45s)
creating index...
index created!
Illegal image file: datasets/camera_0525/mytrain/09269_2.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/10029_4.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/09171_6.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/18941_6.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/09171_8.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/09171_9.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/10029_10.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/09217_10.jpg, and it will be ignored
Illegal image file: datasets/camera_0525/mytrain/18941_10.jpg, and it will be ignored
Found an invalid bbox in annotations: im_id: 7609, area: 0.0 x1: 387.0, y1: 113.0, x2: 391.0, y2: 113.0.
Found an invalid bbox in annotations: im_id: 8206, area: 0.0 x1: 552.0, y1: 54.0, x2: 553.0, y2: 54.0.
Found an invalid bbox in annotations: im_id: 9019, area: 0.0 x1: 301.0, y1: 398.0, x2: 303.0, y2: 398.0.
Found an invalid bbox in annotations: im_id: 9019, area: 0.0 x1: 298.0, y1: 398.0, x2: 305.0, y2: 398.0.
Found an invalid bbox in annotations: im_id: 12643, area: 0.0 x1: 1091.0, y1: 26.0, x2: 1092.0, y2: 26.0.
Found an invalid bbox in annotations: im_id: 13527, area: 0.0 x1: 615.0, y1: 135.0, x2: 616.0, y2: 135.0.
Traceback (most recent call last):
  File "tools/train.py", line 123, in <module>
    main()
  File "tools/train.py", line 119, in main
    run(FLAGS, cfg)
  File "tools/train.py", line 98, in run
    trainer.train(FLAGS.eval)
  File "/home/lthpc/clw/ppdet/engine/trainer_v1.py", line 299, in train
    outputs = model(data)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 898, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/fluid/dygraph/parallel.py", line 578, in forward
    outputs = self._layers(*inputs, **kwargs)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 898, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/lthpc/clw/ppdet/modeling/architectures/meta_arch.py", line 23, in forward
    out = self.get_loss()
  File "/home/lthpc/clw/ppdet/modeling/architectures/cascade_rcnn.py", line 113, in get_loss
    rpn_loss, bbox_loss, mask_loss = self._forward()
  File "/home/lthpc/clw/ppdet/modeling/architectures/cascade_rcnn.py", line 75, in _forward
    body_feats = self.backbone(self.inputs)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 898, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/lthpc/clw/ppdet/modeling/backbones/resnet_v3.py", line 573, in forward
    conv1 = self.conv1(x)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 898, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/fluid/dygraph/container.py", line 97, in forward
    input = layer(input)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 898, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/lthpc/clw/ppdet/modeling/backbones/resnet_v3.py", line 122, in forward
    out = self.conv(inputs)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 898, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/nn/layer/conv.py", line 667, in forward
    use_cudnn=self._use_cudnn)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/nn/functional/conv.py", line 114, in _conv_nd
    pre_bias = getattr(core.ops, op_type)(x, weight, *attrs)
KeyboardInterrupt
/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
I0528 13:07:08.244606 44038 gen_comm_id_helper.cc:181] Server listening on: 127.0.0.1:38975 successful.
I0528 13:07:08.695708 44038 nccl_context.cc:74] init nccl context nranks: 6 local rank: 5 gpu id: 9 ring id: 0
W0528 13:07:11.971997 44038 device_context.cc:404] Please NOTE: device: 9, GPU Compute Capability: 6.1, Driver API Version: 11.2, Runtime API Version: 10.2
W0528 13:07:11.978613 44038 device_context.cc:422] device: 9, cuDNN Version: 7.6.
/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/paddle/tensor/creation.py:125: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
loading annotations into memory...
Traceback (most recent call last):
  File "tools/train.py", line 123, in <module>
    main()
  File "tools/train.py", line 119, in main
    run(FLAGS, cfg)
  File "tools/train.py", line 85, in run
    trainer = Trainer(cfg, mode='train')
  File "/home/lthpc/clw/ppdet/engine/trainer_v1.py", line 67, in __init__
    self.dataset, cfg.worker_num)
  File "/home/lthpc/clw/ppdet/data/reader.py", line 143, in __call__
    self.dataset.parse_dataset()
  File "/home/lthpc/clw/ppdet/data/source/coco.py", line 45, in parse_dataset
    coco = COCO(anno_path)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/site-packages/pycocotools/coco.py", line 85, in __init__
    dataset = json.load(f)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/json/__init__.py", line 296, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/lthpc/anaconda3/envs/ShawnD/lib/python3.7/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
KeyboardInterrupt
